Reinforcement Learning with Model Predictive Control for Highway Ramp Metering

In the backdrop of an increasingly pressing need for effective urban and highway transportation systems, this work explores the synergy between model-based and learning-based strategies to enhance traffic flow management by use of an innovative approach to the problem of highway ramp metering control that embeds Reinforcement Learning techniques within the Model Predictive Control framework. The control problem is formulated as an RL task by crafting a suitable stage cost function that is representative of the traffic conditions, variability in the control action, and violations of a safety-critical constraint on the maximum number of vehicles in queue. An MPC-based RL approach, which merges the advantages of the two paradigms in order to overcome the shortcomings of each framework, is proposed to learn to efficiently control an on-ramp and to satisfy its constraints despite uncertainties in the system model and variable demands. Finally, simulations are performed on a benchmark from the literature consisting of a small-scale highway network. Results show that, starting from an MPC controller that has an imprecise model and is poorly tuned, the proposed methodology is able to effectively learn to improve the control policy such that congestion in the network is reduced and constraints are satisfied, yielding an improved performance compared to the initial controller.Comment: 14 pages, 10 figures, 3 tables, submitted to IEEE Transactions on Intelligent Transportation System

Proactive Emergency Collision Avoidance for Automated Driving in Highway Scenarios

Uncertainty in the behavior of other traffic participants is a crucial factor in collision avoidance for automated driving; here, stochastic metrics should often be considered to avoid overly conservative decisions. This paper introduces a Stochastic Model Predictive Control (SMPC) planner for emergency collision avoidance in highway scenarios to proactively minimize collision risk while ensuring safety through chance constraints. To address the challenge of guaranteeing the feasibility for the emergency trajectory, we incorporate nonlinear tire dynamics in the prediction model of the ego vehicle. Further, we exploit Max-Min-Plus-Scaling (MMPS) approximations of the nonlinearities to avoid conservatism, enforce proactive collision avoidance, and improve computational efficiency in terms of performance and speed. Consequently, our contributions include integrating a dynamic ego vehicle model into the SMPC planner, introducing the MMPS approximation for real-time implementation in emergency scenarios, and integrating SMPC with hybridized chance constraints and risk minimization. We evaluate our SMPC formulation in terms of proactivity and efficiency in various hazardous scenarios. Moreover, we demonstrate the effectiveness of our proposed approach by comparing it with a state-of-the-art SMPC planner and validate the feasibility of generated trajectories using a high-fidelity vehicle model in IPG CarMaker.Comment: 13 pages, 10 figures, submitted to IEEE Transactions on Control Systems Technolog

Distributed control of interconnected systems and its application in traffic control

Over the past few decades, the need for mobility and road transportation has been significantly increased over the world. This turned as a dilemma in many metropolitan areas as traffic congestion costs energy and takes time of all individuals who live in big cities. The exigency for advanced solutions arose when constructing new roads and infrastructures lost its eligibility due to e.g. financial and environmental cost. It then challenges scientists and traffic engineers to elaborate more accurate traffic models and more powerful intelligent transportation systems. The content of this thesis can be separated in two parts. The first part of the thesis focuses on traffic modelling and precisely on suggesting a proper method for describing traffic anomalies like traffic incidents. By introducing two parameters named as incident parameters, the well-known METANET model has been parametrised to describe the effect of traffic anomalies. Simulation and real traffic data has been employed in validating the idea. As the size of the system grows, collecting traffic data, analysing them and making decision become challenging due to scalability. Hence, the concept of distributed control seems a promising direction for analysis and synthesis of interconnected systems like traffic systems. Hence, the second part which is the major part of this thesis as well, deals with the synthesis of discrete time interconnected systems. Such systems are composed of smaller units called subsystems or agents. We let the subsystems belong to the wide class of Linear Parameter Varying systems with the rational dependancy on parameters. For such systems, we aim at designing a distributed control with induced L2 norm minimization chosen as performance requirement. Two types of control strategies are suggested in this thesis {i) scheduled state-feedback control and {ii) scheduled dynamic output feedback control. In both cases, a copy of subsystems' parameters and interconnection structure is used to schedule the controllers. Moreover, we incorporate the saturation of the control input in the distributed control framework. For that, we introduce a special structure in the controller matrices and we enforce this special structure in the synthesis procedure as well

Distributed control of interconnected systems and its application in traffic control

Over the past few decades, the need for mobility and road transportation has been significantly increased over the world. This turned as a dilemma in many metropolitan areas as traffic congestion costs energy and takes time of all individuals who live in big cities. The exigency for advanced solutions arose when constructing new roads and infrastructures lost its eligibility due to e.g. financial and environmental cost. It then challenges scientists and traffic engineers to elaborate more accurate traffic models and more powerful intelligent transportation systems. The content of this thesis can be separated in two parts. The first part of the thesis focuses on traffic modelling and precisely on suggesting a proper method for describing traffic anomalies like traffic incidents. By introducing two parameters named as incident parameters, the well-known METANET model has been parametrised to describe the effect of traffic anomalies. Simulation and real traffic data has been employed in validating the idea.As the size of the system grows, collecting traffic data, analysing them and making decision become challenging due to scalability. Hence, the concept of distributed control seems a promising direction for analysis and synthesis of interconnected systems like traffic systems. Hence, the second part which is the major part of this thesis as well, deals with the synthesis of discrete time interconnected systems. Such systems are composed of smaller units called subsystems or agents. We let the subsystems belong to the wide class of Linear Parameter Varying systems with the rational dependancy on parameters. For such systems, we aim at designing a distributed control with induced L2 norm minimization chosen as performance requirement. Two types of control strategies are suggested in this thesis {i) scheduled state-feedback control and {ii) scheduled dynamic output feedback control. In both cases, a copy of subsystems\u27 parameters and interconnection structure is used to schedule the controllers. Moreover, we incorporate the saturation of the control input in the distributed control framework. For that, we introduce a special structure in the controller matrices and we enforce this special structure in the synthesis procedure as well

Incident Traffic Flow Models

In the trend toward civilization, transportation has always been considered asan indisputable aspect. However, soon it turned out to be a dilemma in manymetropolitan areas how to assess the increased demand in transportation. The exigency for advanced alternative arises when constructing new roads and infrastructures lost its eligibility due to i.e. financial cost. It challenges scientists and traffic engineers then to elaborate more and more powerful intelligent transportation systems i.e. advanced road traffic management/supervision/control solutions.The model-based analysis and synthesis of traffic system require mathematical abstractions of the real traffic in order to properly predict its behavior.One of the most emerging direction is to create ITS solutions resilientto off-nominal traffic conditions, i.e. to traffic incidents. To embed resilienceinto road traffic control algorithms, proper modeling and reconstruction of traffic phenomenon are indispensable. Hence, the first part of this thesis focuses on proper description of incident modeling. Two different nominal traffic flow models namely Aw-Rascle and PW models are chosen. Within these modeling frameworks, incident parameters are properly introduced to describe the effect of traffic anomalies. To consolidate the idea, the microscopic interpretations of this parametrization has been presented. Simulation and real-measured traffic data based model validation is presented through joint state-parameter estimation scheme.The second part of the thesis is devoted to synthesis of an appropriate controlstrategy. We introduce scheduled robust optimization solution using ramp meter, which by encountering real-time incident parameter information, minimizes the effect of demand changes on predefined performance output

Freeway traffic incident reconstruction – A bi-parameter approach

The paper suggests a novel alternative to generalized traffic incident descriptions within the macroscopic traffic model framework. The contribution of the paper is twofold. First, by extending already existing second order macroscopic conservation laws to characterize off-nominal traffic conditions, we define two main incident parameters such as direct and indirect ones. Physical interpretations of this incident parametrization is provided. These incident indicators are relative in view of the nominal traffic flow model parameters and carries physically meaningful macroscopic content. Second, the paper proposes to use a constrained and nonlinear, joint traffic state- and incident parameter reconstruction method and validates the suggested modeling idea via real traffic measurements fitting. Evaluation of the numerical results demonstrate the effectiveness of the methodology

Distributed dynamic output feedback control for discrete-time linear parameter varying systems

Proposed in this note, is a method for scheduled distributed dynamic output feedback controller design. The underlying large-scale system is assumed to be the interconnection of Linear Parameter Varying (LPV) discrete time sub-systems. Following the concept of Integral Quadratic Constraints, robust LPV controller is developed aiming at L2 norm minimisation. The interconnection of the controller has been selected to be identical to the spacial distribution of the sub-systems to secure the level of sparsity in communication topology. By using agentwise full block multipliers in the design phase, distributed output feedback controller design framework is obtained by the sequential use of elimination and dualization lemmas. In order to show the benefits of the suggested methodology, numerical simulation tests are carried out to control the traffic flow in a motorway segment by means of on-ramp input flow gating

Das Gesetz des iterierten Logarithmus fuer kumulative Prozesse

SIGLEDEGerman

Distributed LPV State-Feedback Control with Application to Motorway Ramp Metering

In this paper, we develop a distributed state-feedback controller synthesis algorithm for a discrete-time LPV system that is composed of the interconnection of several subsystems each scheduled by its own parameters. A set of LMI conditions are derived for robust L2-gain performance of such a system in the framework of multiplier-based LPV synthesis. The results have been oriented to be applied in traffic flow control in motorways by ramp metering. First, with the use of a proper transformation, the nonlinear traffic flow model has been represented as the interconnection of LPV subsystems. Then the developed synthesis results have been used to design a gain-scheduled distributed state-feedback controller that keeps the density of all segments around a desired value by the use of ramp metering

Distributed LPV state-feedback control under control input saturation

Developed in this note is a scheduled state-feedback controller synthesis method for discrete-time Linear Parameter Varying (LPV) systems subjected to control input saturation constraints. The static state-feedback gain is scheduled with an exact replica of the parameter matrix. The saturation effect is modeled by introducing time-varying parameters as functions of the control inputs, which are also used to schedule the controller. The synthesis method is then specialized to distributed state-feedback by imposing a particular structure on the feedback gain matrix. An explicit formula is also derived for the computation of the distributed control input from a nonlinear equation. The viability of the proposed method is tested in a simulation environment, for a ramp meter traffic flow control problem